Hello Jolene!

Alexey Kargin is here! Nice to meet you again! Unfortunately, Olga cannot review your project now, so that I will do that. I've added all my comments on your project to cells with different coloring. Please, do not move or delete them - it helps us move faster:

Reviewer's comment v.1 👍 I am using green for my comment if everything is done successfully.
Reviewer's comment v.1 📝 I am using yellow for my comment if I can give you a recommendation or think that something can be improved. These are optional recommendations, but it will be perfect if you work with them in this or future projects.
Reviewer's comment v.1 ✍ I am using red for my comment if the block requires some extra work or corrections. The project couldn’t be accepted with red comments.

To make a dialog, let’s use this style for your remarks:

Student's comment Just like this. To make a similar block - double click on the block, copy and edit it in places you are changing. Also, if you have any questions, you could highlight them using this block.

Now, let’s go to data analysis!


General feedback v.1 Jolene, thank you for sending the project 👍 You've done a good job! I see you demonstrated solid skills in data analysis. You are able to apply any tool you need: python coding, data processing, hypotheses testing, detailed markdown text, and visualization. However, few things need to be corrected in your project. You will find them as red comments. I believe you can easily fix it! Keep up the good work, and come back soon!
Best regards,
Alexey

Hello Jolene!

My name is Olga. I'm happy to review your project today.

When I will see mistake at the first time, I will just point it out. I let you find it and fix it by yourself. I'm trying to prepare you to work as an Data Analyst. SO, at a real job, your team lead will do the same. But if you can't handle this task yet, I will give you a more accurate hint at the next iteration.

Below you will find my comments - please do not move, modify or delete them.

You can find my comments in green, yellow or red boxes like this:

Reviewer's comment Success. Everything is done succesfully.
Reviewer's comment Remarks. Some recommendations.
Reviewer's comment Needs fixing. The block requires some corrections. Work can't be accepted with the red comments.

You can answer me by using this:

Student answer. Text here.
Overall reviewer's comment Jolene, thank you for sending your project. You've done a really good job on it! But, unfortunately, your project is not quite ready for review. This is because the code doesn’t run because of errors. Could you use Kernel-Restrat&Run All for checking cells? Some of them has strange modules and variables. Also could you check the way for reading datasets? You've already done a lot, I am sure you'll take care of it! :)
Student answer. Hi Olga, thank you for reviewing my project. I had created this on my personal jupyter notebook server. I've uploaded the .csv files to the platform. Hopefully it will work now.
Reviewer's comment v.1 👍 Thank you for considering this issue. The upload of the .cvs files is a good idea, but unfortunately, it does not work for me because JupyterHub creates an environment for each user separately, and I cannot see your additional files. It is better to use a path from the brief project information in this case. Also, we could use the `try-except` block here - https://www.w3schools.com/python/python_try_except.asp ```python try: marketing_events_orig = pd.read_csv('ab_project_marketing_events_us.csv') users_orig = pd.read_csv('final_ab_new_users_upd_us.csv') user_events_orig = pd.read_csv('final_ab_events_upd_us.csv') participants_orig = pd.read_csv('final_ab_participants_upd_us.csv') except: marketing_events_orig = pd.read_csv('/datasets/ab_project_marketing_events_us.csv') users_orig = pd.read_csv('/datasets/final_ab_new_users_upd_us.csv') user_events_orig = pd.read_csv('/datasets/final_ab_events_upd_us.csv') participants_orig = pd.read_csv('/datasets/final_ab_participants_upd_us.csv') ```
Student answer. Hi Alexey, thank you for reviewing my project. I didnt understand this nuance of JupyterHub, so thank you for explaining that to me! I've provided updates to your review below. I want to be a good coder, so I appreciate your feedback. I hope I've updated as you expected. Thank you, Jolene G.

General feedback v.2 Jolene, thank you for your work and for improving the project. You've done a really good job! However, I have a few tiny corrections to help you with the stat test part. Also, I have left some additional comments. If you have no questions and everything is Ok, please tell me, and I will accept your project.
Best regards,
Alexey

Student answer. Hi Alexey, thank you for providing explanations and clarification! I included changes for the funnel and bonferonni correction. The last time we used the bonferonni correction, is exactly how I used it this time and my error was not caught, so I'm glad you caught it so I dont make that mistake again. And now I have a better understanding of the calculations. However, I am very stuck on understanding the duplicates and proportions. I tried to breakdown your code to understand what each part is doing but I dont know what the point of it is. I've asked some clarificatin questions in that section. - What may bring the most clarification is understanding what the question being asked is. Thank you, Jolene G.

General feedback v.3 Jolene, thank you for your questions and additional work with the project! Now your project is a true "A" Congratulations! Please find additional comments on your questions. It was pleasure to review your project. Good luck with the next project!
Best regards,
Alexey

Final Project: A/B Testing

Project Description

Technical description

Project Goal

Hypotheses

Tasks

Important things to consider

- The data stopped being collected 11 days after the test period - thus anyone who signed up after Dec 18th will not provide a full 14 days of data

Despcription of the data

Instructions for completing the project

Step 1. Open the data file(s) and examine the general information

Step 2. Data Preprocessing

Step 3. Carry out exploratory data analysis (EDA)

Step 4. Evaluate the A/B test results

Step 5. Conclusion and Recommendations

Reviewer's comment v.1 👍 Perfect introduction with all necessary information!

Import Libraries

Reviewer's comment v.1 👍 Good! It's a nice practice to make all imports at the start of the project code! That is in line with [PEP 8 -- Style Guide for Python Code](https://www.python.org/dev/peps/pep-0008/)

Step 1. Open the data file and read the general information

Plan of Action

Open and save original data as a dataframe

Student answer. V2: Updated .csv uploading START and END
Reviewer's comment v.2 👍 Correct! Thank you!

Examine general info & Take a closer look at specific columns

Marketing Events

Marketing events conclusion

Reviewer's comment v.1 👍 Well done! Please note that we have studied the A/B test results aimed at users from the EU region. In this case, we should consider only the first marketing event.

Users data

Users data conclusion

User Events

User Event Conclusion

Group Participants

Group Participants Conclusion

Step 1 Conclusion

Open the data file and read the general information

Marketing events conclusion

Users data conclusion

User Event Conclusion

Group Participants Conclusion

Reviewer's comment v.1 👍 Well done! A good job at this stage: the data have been loaded and observed in detail.

Step 2. Data Preprocessing

Plan of Action

read and save a working version of data

Student answer. V2: Updated .csv uploading START and END
Reviewer's comment v.2 👍 Correct! Thank you!

Change column names

Change datatypes

Process missing data

Step 2. Conclusion

Data Preprocessing

Actions Performed

Reviewer's comment v.1 👍 Well done! The data have been prepared for the following investigation correctly! Let's go to EDA stage!

Step 3. Carry out exploratory data analysis (EDA)

Examine duplicate users in the test groups

Reviewer's comment v.1 ✍ Based on the `participants` data, we have information about two A/B tests: `recommender_system_test` and `interface_eu_test`. in this case, firstly, we should check duplicate users in both A/B tests and after that check the duplicate users within each A/B test separately. Please correct this part of the project.
Student answer. V2 Update: examine duplicates START - I am interpreting your comment in the following way: - check for duplicate `user_id`s in `recommender_system_test` ***and*** `interface_eu_test` combined - check for duplicate `user_id`s in `recommender_system_test` ***or*** in `interface_eu_test` using separate tables for each test ***Actions*** - I re-evaluated the `participants` table and show that there ar 887 `user_id` duplicates in the combination of the two `ab_test`s - All of the duplicate `user_id`s are in the `interface_eu_test` - This is what I showed before and removed these duplicates - I separated the tests into different queries and looked for duplicate `user_id`s but none were found in either test, including the `interface_eu_test` - this doesnt make sense and I dont know why
Student answer. V2 Update: examine duplicates END
Reviewer's comment v.2 👍 Thank you for detailed checking this moment. In this case, we could also study the issue in a different way:
Reviewer's comment v.2 👍 We found that 887 users took two tests at the same time. It means that 887 users from the `recommender_system_test` test could be affected by the `interface_eu_test` test. Let's check them in the `recommender_system_test` test:
Reviewer's comment v.2 👍 Above we could see how the users from the second test divided into groups of the `recommender_system_test` test. After that, we could check the proportion of the users from the group B of the second test
Reviewer's comment v.2 👍 As we can see above the proportions of the users from group B of the second `interface_eu_test` in both A and B group of the `recommender_system_test` are close. It is mean that the second test affected the test (B) and control (A) groups equally, therefore, most likely will not affect the `recommender_system_test` result.
Student answer. - I am very confused about this part of the project - What is the question being considered here? - Can you please be more clear about what ab test you are talking about - as in please reference the `interface_eu_test` or the `recommender_system_test` - what is the "second test"? Is it the `interface_eu_test` or the `recommender_system_test`? - `user_from_sec_test` which test is this column refering to? I think: `recommender_system_test` - `user_from_sec_test_B_group` which test is this column refering to? again I think `recommender_system_test` but I'm not sure... - I dont understand what you will do with the proportion information now. - The `recommender_system_test` isnt the focus of the study, so why are you focusing on this ab test? - Why are you only looking at Group B?
Reviewer's comment v.3 👍 I am sorry that my comment confused you. Let's check the data that we have: 1. The data contain two A/B tests: - `recommender_system_test` - `interface_eu_test` 2. Each test contains `control - A` and `test or experimental B` groups. Usually, users within `group A` use the service without any new functions. 3. In our case, users who were in the B group of: - `recommender_system_test` used the service with new system recommendation; - `interface_eu_test` used the service with the new interface; 4. It is important to know when we provide two A/B simultaneously if any users took part in several tests. 5. We have checked that and suggested that there were 887 users who took parts in two tests. 6. If we study the results of one of the tests, for example, `recommender_system_test`, it is important to know how many users were in the B group of the `interface_eu_test` test because these users used service with the new version of the interface that could influence on the conversion. In this case, it is important to check the distribution of these users within the test that we have studied; in our example, this is the `recommender_system_test` test. 7. We could see these users distributed evenly in the two groups of the `recommender_system_test` test. It means that about 11% of users within group A and group B of the `recommender_system_test` test used the service with the new version of the interface. 8. After that we could suggest that if `interface_eu_test` test affected on the users convrsions wihtin the `recommender_system_test` was equally the same for group A and B of the `recommender_system_test` test. In this case, we could keep these users within the test.

Conclusion: Examine duplicate users in the test groups

Remove duplicate users from the tests

Does the data contain 15% of new users from the EU region?

Conclusion: Does the data contain 15% of new users from the EU region?

Reviewer's comment v.1 👍 Well done!
Reviewer's comment v.1 ✍ Please study the share of new users from the `EU` region within the `recommender_system_test` test.
Student answer. V2: EU region update START ***Actions*** - Calculated the number and percentage of new users from the EU in the `recommender_system_test` - 4% of the new users are from the EU region
Student answer. V2: EU region update END
Reviewer's comment v.2 👍 Good! Thank you!

Review the first and last data entry collected in data set

Does the data cover the appropriate number of days?

Reviewer's comment v.1 👍 Good! It seems reasonable.

Conclusion: Does the data cover the appropriate number of days?

Reviewer's comment v.1 👍 Correct!

Study conversion at different funnel stages

filter events for users that have 14 days worth of data

Merge the events and test participant tables

Reviewer's comment v.1 👍 Well done! The merged data have been prepared.

Examine the number of events

Conclusion: Examine the number of events

Reviewer's comment v.1 ✍ You have provided a correct description of the event funnel, whereas the `product_cart` is optional. It means that we have a simple product funnel, and in this case, we should calculate the total conversion of all users into each stage. Also, it is not necessary to omit the data for the `product_cart` event in this case. Here we could check if the `product_page` is an optional event for the purchase one. In addition, based on the technical description, we should study the conversion of all registered users into three main stages, and the `product_cart` is one of them. Please return the `product_cart` event to the general order.
Student answer. V2: return `product_cart` returned START - continued through out following results ***Actions*** - returned the optional product_cart stage to the analysis

Remove the optional event

Create funnels for each group without optional stage

Reviewer's comment v.1 👍 Correct! It is reasonable to study the results of the `interface_eu_test` test in this case.

Graph the event funnel for the EU test data

Conclusion: Create funnels for each group without optional stage

Reviewer's comment v.1 ✍ Please add the `product_cart` event into the funnels for A and B groups and describe the results. At this stage, we could also calculate the conversion into main stages.
Student answer. V2 Conversion of Stages START
Reviewer's comment v.2 👍 Well done! Thank you for your additional job here! Please note that now in some cases, the purchase event is ordered before the product cart one. It is not so correct because it seems that the product cart is the users' primary goal. So it is better to reorder the funnel events into a logical order. Alsom we could show the funnels for two groups within single plot: https://plotly.com/python/funnel-charts/#stacked-funnel-plot-with-gofunnel
Student answer. V3 Joined Funnel in correct order START and END - I added a joined funnel with the correct funnel stage order, this is a much more effective way to display the information. Thank you for the information!
Reviewer's comment v.3 👍 Correct! Please do not forget to set the graph title.
Student answer. V2 Conversion of Stages END - A 10% increase in conversion from stage to stage was expected for Group B, however, the results between Group A and B are roughly the same, give or take a few percentage differences. - In fact the trend shows that Group B has a lower conversion rate from stage to stage. This could be due to Group B having slightly fewer participants
Reviewer's comment v.2 👍 Correct! Thank you!

Is the number of events per user distributed equally in the samples?

Conclusion: Is the number of events per user distributed equally in the samples?

Reviewer's comment v.1 👍 Correct!

Are there users who enter both samples?

Reviewer's comment v.1 👍 Well done! However, please check the results after work with my comments within this part of the project.

How is the number of events distributed by days?

Conclusion: How is the number of events distributed by days?

Reviewer's comment v.1 👍 Good job here! The results are correct. Two groups show similar results.

Think of the possible details in the data that you have to take into account before starting the A/B test?

Reviewer's comment v.1 📝 Could you please explain why the z-criterion test requires <= 1% difference between sample sizes?
Student answer. V2: This is important because we want to make sure our test populations are roughly equal when running statistical analysis
Reviewer's comment v.2 👍 Well, when we provide the z-proportion test the number of participants in the groups become important. If one of the groups has a small number of participants, the statistical power of the test may decrease. [Please see additional information](https://practicum.yandex.com/learn/data-analyst/courses/0c2316ef-7b87-49ac-8ff7-f4962da5a0e6/sprints/3246/topics/2a4d7599-c5d5-40fb-87ed-7b67f9ddca1f/lessons/e12e5ccf-b4e4-41ae-a57c-b82012b93557/)

Step 3 Conclusion

Carry out exploratory data analysis (EDA)

Actions Performed

Reviewer's comment v.1 👍 Overall, you have provided an excellent job at this stage. However, please check the conclusions after solving obtained issues.
Reviewer's comment v.2 👍 Thank you!

Step 4. Evaluate the A/B test results

Test Hypothesis

Reviewer's comment v.1 👍 Well done! The hypotheses have been formulated correctly!
Student answer. V2: Removed hypothesis testing of the combined `interface_eu_test` and `recommender_system_test` START
Reviewer's comment v.1 ✍ Please note that we cannot use the mixed test results in this case because two tests should have different designs. Please, omit this part and keep the results for the `interface_eu_test` only. As well as add the `product_card` event.
Student answer. V2: Removed hypothesis testing of the combined `interface_eu_test` and `recommender_system_test` END

Step 4 Conclusion

Actions Performed

Reviewer's comment v.1 👍 Well done! The test has been provided correctly, and the results have been interpreted right! There is not a significant difference between the proportions of experiments.
Reviewer's comment v.1 📝 Please note that we performed several dependent statistical tests simultaneously. In this case, we should use a multiple-comparison correction for the alpha value here. [Please see for detail](https://practicum.yandex.com/learn/data-analyst/courses/0c2316ef-7b87-49ac-8ff7-f4962da5a0e6/sprints/3246/topics/2a4d7599-c5d5-40fb-87ed-7b67f9ddca1f/lessons/4c53ac16-c8cc-4ae1-b4df-118dbc02011b/). The most common method to correct the alpha level is a [Bonferonni Correction](https://www.statisticshowto.com/probability-and-statistics/statistics-definitions/post-hoc/#PHbonferroni).
Student answer. V2: Bonferonni Correction START - use the Bonferroni correction to reduce the family-wise error rate (FWER), i.e. lower the chances of a false result in multiple comparisons - the probability of at least one of a muliti-wise comparison yielding a falst positive result: - alpha (a) = 1 - (1-a)^k = (probability of at least one false positive * 100)% - Given a total number of hypotheses m = 1, FWER = ɑ - Bonferroni correction: alpha (a) / number of hypothesis (m) = significance level for each test
Student answer. V3: Bonferonni Correction - performed correctly START
Student answer. V2: Bonferonni Correction END - With the Bonferonni Correction of 0.18549375000000012, there is not a significant difference between groups A and B for the `login`, `product_cart`, and `product_page` stages. - There is a significant difference for the `purchase` stage - While this may be True, I do not believe it is the 10% difference that was expected - So there is one false negative with out the correction
Reviewer's comment v.2 📝 Overall, you have provided good jobs here, and the results are correct. However, please note that we have tested four H0 hypotheses (for each conversion) in the test above, so in this case, The alpha level with Bonferroni correction is: `0.05 / 4 = 0.0125`. The `FWER` is not the new alpha. It is the probability of making one or more false discoveries or type I errors when performing multiple hypotheses tests and not the alpha level. When we have four tests and alpha as 0.05, there is an 18.55% probability of at least one false-positive result out of 4 tests (as you correctly mentioned above). But the 0.18 is not new alpha. If we use the Bonferroni correction, the alpha will be 0.0125. And now we could estimate the `FWER` for this alpha:
Reviewer's comment v.2 📝 With new alpha = 0.0125 there is an 4.9% probability of at least one false-positive result out of 4 tests. It is very close to 5% probability when we use the alpha = 0.05 for sing test. So in this case, we should uset the corrected alpha as 0.0125:
Student answer. V3: Bonferonni Correction - performed correctly END - I understand the distinction of these values now, thank you for the explanation - I reran the tests with the new alpha = 0.0125 and all of the tests support the null hypothesis, that there is not a statistically significant difference between groups A and B per user funnel stage. - This provides a slight update to the V2 results, which showed that there is a difference in the purchase stage between groups A and B. Although this result occured it did not affect my recommendations from V1, and V3 results are no different - My original recommendations remain the same
Reviewer's comment v.3 👍 Well done! Thank you! The results are correct!

Step 5 Conclusion and Recommendations

Project Description and Goal

Review of the Technical description

Project Conclusions

Recommendations

Reviewer's comment v.1 👍 Good job! Please check the conclusion after revising the main project results, if necessary.